I fixed the problem in a similar way 9ba91d8

Is the tokenizer wrong (remove dash from token characters?)

The rapidfuzz tokenizer only splits at spaces, not at other characters. This is the relevant source code.

Regarding your fix: I think it confuses the following pairs of licenses:

AFL-1.2 AFL-2.1
GPL-1.0 GPL-1.0+
GPL-2.0 GPL-2.0+
GPL-3.0 GPL-3.0+
LGPL-2.0 LGPL-2.0+
LGPL-2.1 LGPL-2.1+
LGPL-3.0 LGPL-3.0+
OLDAP-1.2 OLDAP-2.1
OLDAP-1.2 OLDAP-2.2.1
OLDAP-2.1 OLDAP-2.2.1
OLDAP-2.2 OLDAP-2.2.2

I obtained this list using a little Python script:

import urllib.request
import json
import re

if __name__ == '__main__':
    with urllib.request.urlopen("https://github.com/spdx/license-list-data/raw/main/json/licenses.json") as url:
        data = json.load(url)
    for i, first_license in enumerate(data["licenses"]):
        first_id = first_license["licenseId"]
        first_bag = set(re.findall(r'\w+', first_id.lower()))
        for second_license in data["licenses"][i+1:]:
            second_id = second_license["licenseId"]
            second_bag = set(re.findall(r'\w+', second_id.lower()))
            if first_bag == second_bag:
                print(first_id, second_id)

Probably we should add a test like this as well.

I now ensured in an additional tests that license IDs are always mapped to themselves

@marcelotrevisani I think the failing tests are not related to the changes I made in this PR

@marcelotrevisani I think the failing tests are not related to the changes I made in this PR

yes, they are not related indeed, gonna take a look and I will try to fix them later.
I am not having much time these days to maintain the project due to work and personal life, but I am trying to keep a look into this :)
Thank you for your contribution!

can you rebase this branch to main please?

Hmm, not sure which parts of this PR are still relevant since @beenje merged #552 which is also targeted at the DEC/BSD confusion.